Image synthesizer

ABSTRACT

A digital camera ( 10 ) includes a CCD imager ( 16 ). A long-time exposure image signal obtained by a long-time exposure of the CCD imager ( 16 ) is applied to a terminal (S 1 ) of a switch (SW 1 ), and a short-time exposure image signal obtained by a short-time exposure of the CCD imager ( 16 ) and a gain adjustment is applied to a terminal (S 2 ) of a switch (SW 2 ). A CPU ( 44 ) controls the switch (SW 1 ) by comparing a Y level of the long-time exposure image signal with a reference value Ys so as to generate a combined image signal. More specifically, the CPU ( 44 ) connects the switch (SW 1 ) with the terminal (S 1 ) when a condition of Y level≦Ys is satisfied, and connects the switch (SW 1 ) with the terminal (S 2 ) when a condition of Y level&gt;Ys is satisfied. The CPU ( 44 ) detects a color deviation degree of an object prior to generation of the combined image signal and reduces the reference value Ys on the basis of a detection result. Therefore, the larger color deviation of the object is, the easier selecting of the short-time exposure image signal is.

TECHNICAL FIELD

[0001] The present invention relates to an image combining apparatusbeing applied to a digital camera. More specifically, the presentinvention relates to an image combining apparatus which generates acombined image signal on the basis of image signals respectivelyobtained by a long-time exposure and a short-time exposure.

PRIOR ART

[0002] In a digital camera, when a shutter button is depressed, a YUVsignal is generated on the basis of an RGM signal obtained by apre-exposure of an image sensor, and an optimal exposure amount isdetermined such that an integral value (luminance evaluation value) of aY signal satisfies a predetermined condition. A main exposure of theimage sensor is performed according to the optimal exposure amount, anda YUV signal based on the RGB signal thus obtained is recorded on arecording medium.

[0003] Furthermore, as a digital camera, there is one, when a dynamicrange extension mode is selected, which performs a long-time exposureand a short-time exposure of the same object and combines with eachother a long-time exposure image signal and a short-time exposure imagesignal to which a gain is applied.

[0004] Output characteristics of an image sensor in the long-timeexposure and the short-time exposure vary as shown in FIG. 20(A) andFIG. 20(B), for example. According to FIG. 20(A), at a time thatbrightness of an object, i.e. intensity of reflected light from theobject reaches L1, an output of the image sensor is saturated. On theother hand, according to FIG. 20(B), at a time that the brightness ofthe object reaches L2 (>L1), an output of the image sensor is saturated.Therefore, a Y level of each of pixels forming a long-time exposureimage signal is compared with a reference value Y1 corresponding to thebrightness L1, and if the long-time exposure image signal is selectedwhen a condition of Y level<Y1 is satisfied and if a short-time exposureimage signal to which a gain N is applied is selected when a conditionof Y level≧Y1 is satisfied, a combined image signal in which a dynamicrange is extended can be obtained as shown in FIG. 21.

[0005] However, if there is a deviation in a color of the object, evenif photographing the object at an optimal exposure amount, a color levelof a specific color is saturated. For example, taking notice of a ratioof an RGB signal at a time of macro-imaging a face of a person (skincolor), an R level is much higher than a G level and a B level. Since aY signal is generated on the basis of the RGB signal, when the ratio ofthe RGB signal is extremely distorted, even if the optimal exposureamount is determined so that the luminance evaluation value satisfies apredetermined condition, the level of the R signal obtained by the mainexposure is saturated. Consequently, the ratio of the RGB signal isdeviated from an original ratio, and therefore, a hue of the image to bereproduced is also distorted.

[0006] A problem of such the distortion of the hue conspicuously occurswhen the object deviated to the specific color is photographed in thedynamic range extension mode. For example, in a case the Y level of asection A forming a face exceeds the reference value Y1, and the Y levelof a section B is equal to or less than the reference value Y1 and the Rlevel thereof is saturated as a long-time exposure image shown in FIG.22(A), if the long-time exposure image is combined with a short-timeexposure image (to which a gain has been applied) shown in FIG. 22(B),an combined image in which only the hue of the section B is distorted asshown in FIG. 22(C) is generated. In this case, the section B isdisplayed so as to be raised, and the distortion of the hue is moreconspicuous than that in the long-time exposure image shown in FIG.22(A).

SUMMARY OF THE INVENTION

[0007] Therefore, it is a primary object of the present invention toprovide a novel image combining apparatus.

[0008] Another object of the present invention is to provide an imagecombining apparatus capable of minimize a distortion of a hue of acombined image signal.

[0009] Further object of the present invention is to provide an imagecombining method capable of minimizing occurrence a distortion in a hueof a combined image signal.

[0010] According to the present invention, an image combining apparatuswhich generates, on the basis of a first image signal of an objectobtained by a first exposure according to a first exposure amount and asecond image signal of the object obtained by a second exposureaccording to a second exposure amount which is less than the firstexposure amount, a combined image signal of the object, comprising: acomparing means for comparing a brightness relating level of any one ofthe first image signal and the second image signal with a referencevalue; a first selecting means for selecting the first image signal whenthe brightness relating level is equal to or less than the referencevalue; a second selecting means for selecting the second image signalwhen the brightness relating level is larger than the reference value; adeviation degree detecting means for detecting a deviation degree ofcolor of the object; and a reducing means for reducing the referencevalue on the basis of the deviation degree.

[0011] When the combined image signal is generated on the basis of thefirst image signal obtained by the first exposure according to the firstexposure amount and the second image signal obtained by the secondexposure according to the second exposure amount, the brightnessrelating level of any one of the first image signal and the second imagesignal is compared with the reference value by the comparing means. Whenthe brightness relating level is equal to or less than the referencevalue, the first image signal is selected by the first selecting means,and when the brightness relating level is larger than the referencelevel, the second image signal is selected by the second selectingmeans. Thus, the combined image signal is generated. On the other hand,the deviation degree detecting means detects the deviation degree of thecolor of the object, and the reducing means reduces the reference levelon the basis of the detected deviation degree.

[0012] Since the second exposure amount is less than the first exposureamount, even if the color of the object is deviated to the specificcolor, a color component of the specific color is hard to saturate, andthe hue of the second image signal is hard to distort. If the referencevalue is reduced according to the detected deviation degree, the secondimage signal becomes easy to be selected, and it is possible to preventa situation in which a distortion occurs in the hue of the combinedimage signal.

[0013] In a case of fetching a third image signal of the object obtainedby an exposure according to a predetermined exposure amount, thedeviation degree can be obtained by detecting a color saturation degreeand a luminance saturation degree of the specific color on the basis ofthe third image signal and subtracting the luminance saturation degreefrom the color saturation degree. When all the color levels of thespecific color are saturated, the luminance level is also saturated.Thereupon, a pixel in which the color level of the specific color issaturated but the luminance level thereof is not saturated is regardedas being deviated to the specific color. Therefore, by subtracting theluminance saturation degree from the color saturation degree, thedeviation degree can be obtained.

[0014] If the number of pixels in which the color level of the specificcolor is saturated is regarded as the color saturation degree, and ifthe number of pixels in which the luminance is saturated is regarded asthe luminance saturation degree, it is possible to accurately and easilycalculate the deviation degree.

[0015] It is noted that the predetermined exposure amount is preferablyless than the first exposure amount and is more than the second exposureamount.

[0016] In one embodiment, the greater the deviation degree is, thelarger the reference value is reduced. Thus, the greater the deviationdegree is, the easier the selection of the second image signal is.

[0017] In another embodiment, it is determined whether or not each of aplurality of sections forming the object image satisfies thepredetermined condition. The reducing means weights the deviation degreedepending upon the number of the sections satisfying the predeterminedcondition and reduces the reference value based on a result of theweighting. The predetermined condition preferably includes a firstcondition indicating that a noticed section is a specific color, and thesecond condition indicating that the noticed section has a highluminance.

[0018] According to the present invention, an image combining methodwhich generates, on the basis of a first image signal of an objectobtained by a first exposure according to a first exposure amount and asecond image signal of the object obtained by a second exposureaccording to a second exposure amount which is less than the firstexposure amount, a combined image signal of the object, comprisingfollowing steps of: (a) comparing a brightness relating level of any oneof the first image signal and the second image signal with a referencevalue; (b) selecting the first image signal when the brightness relatinglevel is equal to or less than the reference value; (c) selecting thesecond image signal when the brightness relating level is larger thanthe reference value; (d) detecting a deviation degree of color of theobject; and (e) reducing the reference value on the basis of thedeviation degree.

[0019] When the combined image signal is generated on the basis of thefirst image signal obtained by the first exposure according to the firstexposure amount and the second image signal obtained by the secondexposure according to the second exposure amount, the brightnessrelating level of any one of the first image signal and the second imagesignal is compared with the reference value. When the brightnessrelating level is equal to or less than the reference value, the firstimage signal is selected, and when the brightness relating level islarger than the reference value, the second image signal is selected.Thus, the combined image signal is generated. Herein, the referencevalue is reduced based on the deviation degree of the color of theobject.

[0020] Since the second exposure amount is less than the first exposureamount, even if the color of the object is deviated to the specificcolor, a color component of the specific color is hard to saturate, anda hue of the second image signal is hard to distort. If the referencevalue is reduced according to the detected deviation degree, the secondimage signal becomes easy to be selected, and it is possible to preventa situation in which a distortion occurs in the hue of the combinedimage signal.

[0021] The above described objects and other objects, features, aspectsand advantages of the present invention will become more apparent fromthe following detailed description of the present invention when takenin conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0022]FIG. 1 is a block diagram showing one embodiment of the presentinvention;

[0023]FIG. 2 is a block diagram showing one example of a signalprocessing circuit applied to FIG. 1 embodiment;

[0024]FIG. 3 is a flowchart showing a part of an operation of a CPUapplied to FIG. 1 embodiment;

[0025]FIG. 4 is a flowchart showing another part of the operation of theCPU applied to FIG. 1 embodiment;

[0026]FIG. 5 is a flowchart showing the other part of the operation ofthe CPU applied to FIG. 1 embodiment;

[0027]FIG. 6 is a flowchart showing a further part of the operation ofthe CPU applied to FIG. 1 embodiment;

[0028]FIG. 7 is a flowchart showing another part of the operation of theCPU applied to FIG. 1 embodiment;

[0029]FIG. 8 is an illustrative view showing one example of a tablestoring an RGB signal and a Y signal;

[0030]FIG. 9 is a block diagram showing another embodiment of thepresent invention;

[0031]FIG. 10 is a block diagram showing one example of a signalprocessing circuit applied to FIG. 9 embodiment;

[0032]FIG. 11 is a flowchart showing a part of an operation of the CPUapplied to FIG. 9 embodiment;

[0033]FIG. 12 is a flowchart showing another part of the operation ofthe CPU applied to FIG. 9 embodiment;

[0034]FIG. 13 is a flowchart showing the other part of the operation ofthe CPU applied to FIG. 9 embodiment;

[0035]FIG. 14 is a flowchart showing a further part of the operation ofthe CPU applied to FIG. 9 embodiment;

[0036]FIG. 15 is a flowchart showing another part of the operation ofthe CPU applied to FIG. 9 embodiment;

[0037]FIG. 16 is a flowchart showing the other part of the operation ofthe CPU applied to FIG. 9 embodiment;

[0038]FIG. 17 is a flowchart showing a further part of the operation ofthe CPU applied to FIG. 9 embodiment;

[0039]FIG. 18 is an illustrative view showing a part of an operation ofFIG. 9 embodiment;

[0040]FIG. 19 is an illustrative view showing another part of theoperation of FIG. 9 embodiment;

[0041]FIG. 20(A) is a graph showing an output of a sensor with respectto brightness of an object when performing a long-time exposure;

[0042]FIG. 20(B) is a graph showing an output of a sensor with respectto the brightness of the object when performing a short-time exposure;

[0043]FIG. 21 is an illustrative view showing a combining process of along-time exposure image signal and a short-time exposure image signal;

[0044]FIG. 22(A) is an illustrative view showing an example of along-time-exposure image;

[0045]FIG. 22(B) is an illustrative view showing an example of ashort-time-exposure image to which a gain is applied; and

[0046]FIG. 22(C) is an illustrative view showing an example of acombined image.

BEST MODE FOR PRACTICING THE INVENTION

[0047] Referring to FIG. 1, a digital camera 10 of this embodimentincludes an optical lens 12 and an aperture member 14. An optical imageof an object is incident onto a receiving surface of the CCD imager(image sensor) 16 through these members. A camera signal correspondingto the incident optical image, i.e., raw image signal is generated by aphoto-electronic conversion on the light-receiving surface. It is notedthat a light-receiving surface is covered with a primary color filterhaving a Bayer array (not shown), and each of pixel signals forming thecamera signal has color information of any one of R, G and B.

[0048] When a power is turned on, a CPU 44 respectively sets an apertureamount and an exposure time period to the aperture member 14 and a TG(Timing Generator) 18, and instructs the TG 18 to perform an exposure atevery 1/15 seconds. The TG 18 exposes the CCD imager 16 at every 1/15seconds and reads the camera signal generated by the exposure from theCCD imager 16. The camera signal of each frame read at every 1/15seconds is applied to an image processing circuit 24 through awell-known noise removal and a level adjustment in a CDS/AGC circuit 20and an A/D conversion in an A/D converter 22.

[0049] The signal processing circuit 24 is constituted as shown in FIG.2. The camera signal is applied with a gain (initial value) in amultiplier 24 a and is subjected to a color separation in a colorseparation circuit 24 a. Each of pixels forming the camera signal onlyhas any one of an R information signal (R signal), a G informationsignal (G signal) and a B information signal (B signal), and therefore,two color information signals lacking in each of the pixels iscomplemented by the color separation circuit 24 b. The R signal, the Gsignal and the B signal forming respective pixels are simultaneouslyoutput from the color separation circuit 24 b. The R signal, the Gsignal and the B signal output at every pixel are applied to a YUVconversion circuit 24 d through a white balance adjustment circuit 24 c,and whereby, an image signal constructed by a Y signal (luminancesignal), a U signal (color difference: R-Y) and a V signal (colordifference: B-Y) is generated.

[0050] Returning to FIG. 1, a switch SW1 is connected to a terminal S2,and the image signal output from the signal processing circuit 24 isapplied to a second memory controller 30 via the switch SW1. The secondmemory controller 30 writes the applied image signal to an image signalstorage area 32 a of a second memory 32.

[0051] A video encoder 34 reads the image signal in the image signalstorage area 32 a via the second memory controller 30, encodes the readimage signal of each frame onto a composite image signal of an NTSCformat and applies the encoded composite image signal to an LCD monitor36. The LCD monitor 36 is displayed with a real-time motion image, i.e.,through image of the object at a ratio of 15 fps.

[0052] When a shutter button 46 is depressed, the CPU 44 instructs theTG 18 to perform a pre-exposure and fetches the RGB signal and the Ysignal generated by the signal processing circuit 24 on the basis of thepre-exposure. The CPU 44 determines a reference value Ys for imagecombining based on the R signal and the G signal, adjusts a whitebalance based on the RGB signal and determines a first exposure timeperiod Sm1 for long-time exposure and a second exposure time period Sm2for short-time exposure based on the Y signal. The determined firstexposure time period Sm1 is longer than an optimal exposure time periodat a time of performing normal photographing (in which a main exposureis performed only once), and the second exposure time period Sm2 isshorter than the optimal exposure time period. Succeedingly, the CPU 44instructs the TG 18 to respectively perform a long-time exposure (mainexposure according to the first exposure time period Sm1) and ashort-time exposure (main exposure according to the second exposure timeperiod Sm2) at noticed two frames. The TG 18 performs the long-timeexposure on the CCD imager 16 at a first frame out of the noticed twoframes, and performs reading of a camera signal generated by thelong-time exposure and performs the short-time exposure on the CCDimager 16 at a second frame. A camera signal generated by the short-timeexposure is read from the CCD imager 16 at a frame successive to thesecond frame.

[0053] The CPU 44, when a long-time exposure image signal (image signalbased on the long-time exposure) is output from the signal processingcircuit 24, instructs a first memory controller 26 to perform writing.The long-time exposure image signal is stored in the first memory 28 bythe first memory controller 26. The CPU 44, when the camera signalgenerated by the short-time exposure is read from the CCD imager 16,further changes the gain of the multiplier 24 a shown in FIG. 2 from aninitial value to a predetermined value N and instructs the first memorycontroller 26 to read the long-time exposure image signal. A short-timeexposure image signal which is based on the short-time exposure and towhich the gain is applied is output from the signal processing circuit24, and the long-time exposure image signal is output from the firstmemory controller 26. The long-time exposure image signal and theshort-time exposure image signal corresponding to the same pixel aresimultaneously applied to a terminal S1 and the terminal S2 of theswitch SW1.

[0054] The CPU 44 further fetches the Y signal of each of pixels formingthe long-time exposure image signal, compares the level of the fetched Ysignal with the reference value Ys at every pixel and controls switchingof the switch SW1 in response to a comparison result. The switch SW1 isconnected to the terminal S1 when a condition of Y signal level≦Ys issatisfied and is connected to the terminal S2 when a condition of Ysignal level>Ys is satisfied. When connecting to the terminal S1, apixel signal forming the long-time exposure image signal is selected,and when connecting to the terminal S2, a pixel signal forming theshort-time exposure image signal is selected, and whereby, a combinedimage signal having a dynamic range extended is generated.

[0055] When the combined image signal is output from the switch SW1, theCPU 44 instructs the second memory controller 30 to perform writing. Thecombined image signal is temporarily stored in the image signal storagearea 32 a of the second memory 32 by the second memory controller 30.The CPU 44 succeedingly instructs a JPEG codec 38 to perform acompression process. The JPEG codec 38 reads the combined image signalstored in the image signal storage area 32 a through the second memorycontroller 30 and performs a compression process complying with a JPEGformat on the read combined image signal. When a compressed image signalis obtained, the JPEG codec 38 applies the generated compressed imagesignal to the second memory controller 30. The compressed image signalis stored in a compressed signal storage area 32 b by the second memorycontroller 30.

[0056] When a storing process of the compressed image signal iscompleted, the CPU 44 reads the compressed image signal from thecompressed signal storage area 32 b through the second memory controller30 and records the read compressed image signal onto a memory card 42.Thus, an image file is created within the memory card 42. It is notedthat the memory card 42 is a detachable nonvolatile recording medium andbecomes accessible by the CPU 44 when being loaded in a slot 40.

[0057] When the power is turned on, a process according to a flowchartshown in FIG. 3 to FIG. 7 is executed by the CPU 44. It is noted that acontrol program corresponding to the flowchart is stored in a ROM 48.

[0058] First, a through image displaying process is performed in a stepS1, and it is determined whether or not the shutter button 46 isoperated in a step S3. While the shutter button 46 is not operated, anAE process for monitoring is performed in a step S5 and then, theprocess returns to the step S1. Thus, an aperture amount set to theaperture member 14 and an exposure time period set to the TG 18 arerepeatedly adjusted, and whereby, the through image having moderatebrightness is displayed on the monitor 36. It is noted that theprocesses in the steps S1 and S5 are executed in response to a VD pulsegenerated from the TG 18 at every 1/15 seconds.

[0059] When the shutter button 46 is operated, an exposure setting formetering is performed in a step S7. More specifically, the exposure timeperiod of 1/1200 seconds is set to the TG 18, and the aperture amount ofa maximum opening is set to the aperture member 14. It is determinedwhether or not the VD pulse is applied from the TG 18 in a step S9, andif “YES”, a pre-exposure for metering is instructed to the TG 18 in astep S11. The TG 18 performs the pre-exposure 1/200 seconds at a currentframe on which the instruction is applied and reads from the CCD imager16 a camera signal generated by the pre-exposure at a next framesuccessive to the current frame. A Y signal based on the read camerasignal is output from the YUV conversion circuit 24 d shown in FIG. 2 atthe frame in which the reading has been performed. Therefore, it isdetermined whether or not the VD pulse is generated in a step S13, andif “YES” is determined, one frame of Y signal is fetched from the YUVconversion circuit 24 d in a step S15. The fetched one frame of Y signalis the Y signal based on the pre-exposure in the step S11.

[0060] In a step S17, an exposure time period Sp and an aperture amountF are calculated on the basis of the fetched Y signal. Morespecifically, a luminance evaluation value Iy is obtained by integratingthe Y signal throughout one frame period, and then, the exposure timeperiod Sp and the aperture amount F in which the luminance evaluationvalue Iy satisfies a predetermined condition are calculated. In a stepS19, the aperture amount F is set to the aperture member 14, and theexposure time period Sp is set to the TG 18. It is noted that theexposure setting in the step S19 is for determining the reference valueYs, the white balance adjustment and the optimal exposure time.

[0061] When the VD pulse is generated after completion of the exposuresetting, “YES” is determined in a step S 21, and the pre-exposure isinstructed to the TG 18 in a step S23. The TG 18 performs thepre-exposure according to the exposure time period Sp and reads thecamera signal generated by the pre-exposure from the CCD imager 16. Whenthe VD pulse is generated after the instruction of the pre-exposure, theprocess proceeds from a step S25 to a step S27 so as to fetch the RGBsignal output from the white balance adjustment circuit 24 c shown inFIG. 2 and the Y signal output from the YUV conversion circuit 24 d.Both of the RGB signal and the Y signal fetched are signals based on thepre-exposure in the step S23. In a step S29, the fetched RGB signal andY signal are stored in a table 44 a shown in FIG. 8. At this time, acommon pixel number is assigned to the RGB signal and the Y signal ofthe same pixel. It is determined whether or not one frame fetching ofthe signal is completed in a step S31, and the processes in the stepsS27 and S29 are repeated until “YES” is determined.

[0062] The reference value Ys is determined on the basis of the fetchedR signal and G signal in a step S33, the gain of the white balanceadjustment circuit 22 is set to an optimal value on the basis of thefetched RGB signal in a step S35, and an optimal exposure time period iscalculated on the basis of the fetched Y signal in a step S37. Theoptimal exposure time period calculated in the step S37 is an exposuretime period which becomes optimal in a case the main exposure isperformed once, and therefore, a time period longer than the optimalexposure time period is determined to be the first exposure time periodSm1 for long-time exposure in a step S39, and a time period shorter thanthe optimal exposure time period is determined to be the second exposuretime period Sm2 for short-time exposure in a step S41.

[0063] The first exposure time period Sm1 is set to the TG 18 in a stepS43, and then, “YES” is determined in a step S45 in response togeneration of the VD pulse. Thereupon, the initial gain is set to themultiplier 24 a shown in FIG. 2 in a step S47, a main exposure isinstructed to the TG 18 in a step S49, and a writing instruction isapplied to the first memory controller 26 in a step S51. The TG 18performs the long-time exposure according to the first exposure timeperiod Sm1 and reads a camera signal thus generated from the CCD imager16. The read camera signal has a characteristic shown in FIG. 20 (A).The signal processing circuit 24 generates a long-time exposure imagesignal on the basis of the camera signal, and the generated long-timeexposure image signal is written to the first memory 28 by the firstmemory controller 26.

[0064] The second exposure time period Sm2 is set to the TG 18 in a stepS53, and then, “YES” is determined in a step S55 in response togeneration of the VD pulse. The gain N is set to the multiplier 24 a ina step S57, the main exposure is instructed to the TG 18 in a step S59,and a reading instruction is applied to the first controller 26 in astep S61. The TG 18 performs the short-time exposure according to thesecond exposure time period Sm2, and a camera signal thus generated hasa characteristic shown in FIG. 20(B). The signal processing circuit 24generates a short-time exposure image signal to which the gain N isapplied on the basis of the camera signal obtained by the short-timeexposure. On the other hand, the first memory controller 26 reads thelong-time exposure image signal from the first memory 28 in response tothe reading instruction.

[0065] Consequently, the long-time exposure image signal and theshort-time exposure image signal are simultaneously applied to theterminals S1 and S2 forming the switch SW1. That is, an X-th pixelsignal forming the short-time exposure image signal is applied to theterminal S2 at the same time that the X-th pixel signal forming thelong-time exposure image signal is applied to the terminal S1.

[0066] A writing instruction is applied to the second memory controller30 in a step S63, and a Y signal is fetched from the first memorycontroller 26 in a following step S65. The fetched Y signal is the Ysignal forming the long-time exposure image signal, and a timing whenthe X-th Y signal is fetched is little faster than a timing when theX-th pixel signal (YUV signal) is applied to the terminal S1. The levelof the fetched Y signal is compared with the reference value Ys in astep S67. Then, if a condition of Y signal level>Ys is satisfied, theprocess proceeds to a step S69, and if a condition of Y signal level≦Ysis satisfied, the process proceeds to a step S71. The switch SW1 isconnected to the terminal S1 so as to select the long-time exposureimage signal in the step S69, and the switch SW1 is connected to theterminal S2 so as to select the short-time exposure image signal in thestep S71. It is determined whether or not one frame of a comparisonprocess is completed in a step S73, and the processes from the steps S65to S71 are repeated until “YES” is determined.

[0067] Thus, one frame of combined image signal having a characteristicshown in FIG. 21 is generated. Prior to switching control of the switchSW 1, the writing instruction is applied to the second memory controller30, and therefore, the generated combined image signal is stored in theimages signal storage area 32 a of the second memory 32 by the secondmemory controller.

[0068] A compression instruction is applied to the JPEG codec 38 in astep S75, and a compressed image signal generated by the JPEG coded 38and held in the compressed signal storage area 32 b of the SDRAM 32 isrecorded onto the memory card 42 in a file format in a step S77. Aftercompletion of such the recording process, the process returns to thestep S1.

[0069] The reference value determining process in the step S33 isexecuted in a subroutine shown in FIG. 7. First, the total number ofpixels in which the R signal level is saturated is detected as Rsat in astep S81, and the total number of pixels in which the G signal level issaturated is detected as Gsat in a step S83. These processes areperformed by comparing a threshold value with each of R signal and Gsignal stored in the table 44 a, and the number of pixels in which an Rlevel exceeds the threshold value and the number of pixels in which a Glevel exceeds the threshold value is rendered the Rsat and Gsat,respectively. The detected Rsat and Gsat are respectively defined ascolor saturation degrees of the R signal and the G signal.

[0070] A difference number of pixels Rsat_N is calculated by subtractingthe Gsat from the Rsat in a step S85. It is determined whether or notthe Rsat_N is larger than “0” in a step S87. If “YES”, the processdirectly proceeds to a step S91, and if “NO”, “0” is set to the Rsat_Nin a step S89 and then, the process proceeds to the step S91.

[0071] The Y signal is generated by weighting and adding of the Rsignal, the G signal and the B signal in the ratio of 3:6:1, andtherefore, the G signal exerts the greatest influence upon the Y signal.Furthermore, when the G signal is saturated, the R signal and the Bsignal are also saturated (i.e., the luminance level is saturated), andthere never occurs a situation that the G signal is saturated while theR signal and the B signal are not saturated. Therefore, the Gsat can bedefined as a luminance saturation degree. Thereupon, the Rsat_Ncalculated in the step S87 can be regarded as the total number of pixelsin which no luminance saturation occurs and the R signal level issaturated, and can be regarded as a deviation degree of the R signal inreference to the G signal.

[0072] It is noted that the processes in the steps S87 and S89 are oftaken into account that there is a possibility that the Rsat_N indicatesa numerical value of minus (−) by an error of the setting in the signalprocessing circuit 24. Furthermore, the reason why the Rsat_N relatingto the R signal is calculated is that the R signal exerts the greatestinfluence upon a color of a skin of a person.

[0073] It is determined whether or not a condition shown in an equation1 is satisfied in a step S91. If the condition is satisfied, the processdirectly proceeds to a step S95 while if the condition is not satisfied,the Rsat_N is renewed according to an equation 2 in a step S93 and then,the process proceeds to the step S95. In the step S95, the referencevalue Ys is determined according to an equation 3.

Rsat_(—) N*K≦Ysmax−Ysmin  [equation 1]

[0074] K: constant

[0075] Ysmax: maximum value which the Ys can take

[0076] Ysmin: minimum value which the Ys can take

Rsat_(—) N*K=Ysmax−Ysmin  [equation 2]

Ys=Ysmax−Rsat_(—) N*K  [equation 3]

[0077] Since the Ysmax and the Ysmin are respectively a maximum valueand a minimum value which the Ys can take, the Ys has to be determinedin the Ysmax to the Ysmin range. According to the equation 3, the Ys isobtained by subtracting the Rsat_N*K from the Ysmax, and therefore, theRsat_N*K has to be equal to or less than “Ysmax-Ysmin” in order to makethe Ys fall within the Ysmax to the Ysmin range. Thus, when thecondition of the equation 1 is not satisfied, the Rsat_N*K is correctedaccording to the equation 2.

[0078] As understood form the above description, when the combined imagesignals is generated on the basis of the long-time exposure image signaland the short-time exposure image signal, the Y signal level of thelong-time exposure image signal is compared with the reference value Ys(S67). Then, when the condition of Y signal level≦Ys is satisfied, thelong-time exposure image signal is selected by the switch SW1 (S69), andwhen the condition of Y signal level>Ys is satisfied, the short-timeexposure image signal is selected by the switch SW1 (S71). Thus, thecombined image signal is generated. The deviation degree of the color ofthe object is detected prior to the switching control of the switch SW1(S81 to S85), and the reference value is reduced on the basis of thedetected deviation degree (S95).

[0079] The deviation degree of the color of the object is obtained byrespectively detecting the color saturation degree of the R level andthe color saturation degree of the G level (=luminance saturationdegree) on the basis of the R signal and the G signal obtained by thepre-exposure (S81, S83) and subtracting the luminance saturation degreefrom color saturation degree (S85). When all the R level, the G leveland the B level are saturated, the Y level is also saturated. Thus, apixel in which the R level is saturated but the Y level is not saturatedcan be regarded as being deviated to red. Therefore, the deviationdegree to red is obtained by subtracting the color saturation degree ofthe G level from the color saturation degree of the R level.

[0080] In a case of the short-time exposure, even if the color of theobject is deviated to a specific color, a color component of thespecific color is hard to saturate, and a hue of the short-time exposureimage signal is also hard to distort. Thus, if the reference value Ys isreduced on the basis of the detected deviation degree, the short-timeexposure image signal is easy to be selected, and it is possible toprevent a situation in which the distortion of the hue in the combinedimage signal occurs.

[0081] It is noted that the Gsat is subtracted from the Rsat at a timeof acquiring the Rsat_N in this embodiment; however, it is alsoappropriate that the total number of the pixels in which the level ofthe Y signal is saturated is detected as Ysat and then, the Ysat issubtracted from the Rsat. Furthermore, although the aperture amounttakes a constant (aperture amount obtained by the pre-exposure) and theexposure time period is varied at a time of performing the long-timeexposure and short-time exposure in this embodiment, the aperture amountmay be varied in addition to or in place of the exposure time period.

[0082] Furthermore, although the reference value Ys is determined on thebasis of the Rsat_N in this embodiment, Bsat_N (=Bsat−Gsat) iscalculated and then, the reference value Ys may be determined on thebasis the Bsat_N. If so, with respect to a color including a largenumber of B signals, it is possible to prevent a situation in which asection distorted in hue is raised.

[0083] In addition, although the reference value Ys is linearly changedin accordance with a value of the Rsat_N in the arithmetic operationaccording to the equation 3, the reference value Ys may be nonlinearly(according to a quadratic function) changed in accordance with the valueof the Rsat_N. Furthermore, although the long-time exposure is performedprior to the short-time exposure in this embodiment, an order reversethereto may be appropriate.

[0084] Also, when acquiring the Rsat_N, the total number of pixels Gsatin which the G level is saturated is subtracted from the total number ofpixels Rsat in which the R level is saturated in this embodiment.However, it is also appropriate that the R level and the G level arecompared with a threshold value pixel by pixel, the number of pixels inwhich the R level is saturated but the G level is not saturated isintegrated, and the integral may be defined as the Rsat_N.

[0085] Referring to FIG. 9 and FIG. 10, the digital camera 10 of anotherembodiment is the same as FIG. 1 embodiment except that counters 44 rand 44 g are formed within the CPU 44 and the processes according to theflowchart shown in FIG. 11 to FIG. 17 are executed by the CPU 44. Inaddition, steps S101 to S131 shown in FIG. 11 and FIG. 12 are the sameas the steps S1 to S31 shown in FIG. 3 and FIG. 4, and steps S155 toS193 shown in FIG. 13 to FIG. 15 are the same as the steps S39 to S77shown in FIG. 4 to FIG. 6. Therefore, a duplicated description isomitted as to a common section.

[0086] The processes in the steps S 127 and S129 shown in FIG. 12 areperformed during one frame period, and if “YES” is determined in thestep S131, the white balance adjustment process is performed in a stepS135, and the optimal exposure time period is calculated in a step S137.The white balance adjustment and calculation of the optimal exposuretime period are performed on the basis of an RGB signal and a Y signalstored in the table 44 a. When the optimal exposure time period iscalculated, the optimal exposure time period is set to the TG 18 in astep S139.

[0087] It is determined whether or not a VD pulse is generated in a stepS141, and if “YES” is determined, a pre-exposure is performed in a stepS143. The pre-exposure is executed according to the aperture amount Fcalculated in the step S117 and the optimal exposure time periodcalculated in the step S137. When the VD pulse is generated after thepre-exposure according to the optimal exposure amount, “YES” isdetermined in a step S145, and processes the same as in the steps S127to S131 are performed in steps S147 to S151. Thus, one frame of the RGBsignal and the Y signal based on the pre-exposure in the step S143 isstored in the table 44 a. If “YES” is determined in the step S151, theprocess proceeds to the step S155 through a reference valuedetermination process in a step S153.

[0088] The reference value determination process in the step S153 isexecuted by a subroutine shown in FIG. 16 and FIG. 17. First, a countvalue Rcnt of the counter 44 r and a count value Gcnt of the counter 44g are set to “0” in a step S201, and Yij, (R−G) ij and (B−G) ij arecalculated with reference to the table 44 a in a step S203.

[0089] An object image is, as shown in FIG. 18, divided into 8 in ahorizontal direction and in a vertical direction, and whereby, dividedareas of 64 are formed on the screen. i and j indicate positions of thedivided areas in the horizontal direction and in the vertical direction,and any one of “1” to “8” is assigned to the i and the j. The Yijindicates an integral value detected from the divided area (i, j) andcan be defined as a luminance evaluation value. On the other hand, the(R−G) ij indicates a difference between integral values respectivelybased on the R signal and the G signal detected from the divided area(i, j), and the (B−G) ij indicates a difference of integral valuesrespectively based on the B signal and the G signal detected from thedivided area (i, j).

[0090] An arithmetic operation of an equation 4 is executed in a stepS205. According to the equation 4, each of the (R−G) ij and the (B−G) ijis divided by the luminance evaluation value.

fy (B−G) ij=(B−G) ij/Yij  [equation 4]

fy (B−G) ij=(B−G) ij/Yij

[0091] Each of the values indicated by the (R−G) ij and the (B−G) ijreflects the exposure amount at a time of performing the pre-exposure.That is, the more the exposure amount is, the larger the numerical valueis while the less the exposure amount is, the smaller the numericalvalue is. When each of the (R−G) ij and the (B−G) ij having such thecharacteristics is defined as a color evaluation value in each ofdivided areas, the color evaluation value varies by the exposure amount.On the other hand, the color of the object is essentially independent ofthe exposure amount and is constant at all times unless the object andthe light source are changed. Accordingly, even if the exposure amountis changed, the color evaluation value should take the same value.Therefore, each of the (R−G) ij and the (B−G) ij is divided by theluminance evaluation value Yij relating to the exposure amount accordingto the equation 4, and a divided value is rendered the color evaluationvalue. Thus, the color evaluation value becomes independent of theexposure amount, and this makes it possible to accurately evaluate thecolor of the object.

[0092] It is determined whether or not the fy (R−G) ij and the fy (B−G)ij are included in an area R of a color distribution map shown in FIG.19 in a step S207, and it is determined whether or not the Yij exceeds athreshold value th1 in a step S209. If “NO” is determined in any one ofthe steps S207 and S209, the process directly proceeds to a step S213while if “YES” is determined in both of the steps S207 and S209, thecount value Rent is incremented in a step S211 and then, the processproceeds to the step S213. Accordingly, the count value Rcnt isincremented when the color evaluation value at a noticed divided areabelongs to the area R, and an image in the divided area has a highluminance.

[0093] It is determined whether or not the fy (R−G) ij and the fy (B−G)ij are included in an area G of the color distribution map shown in FIG.19 in the step S213, and it is determined whether or not the Yij exceedsa threshold value th2 in a step S215. If “NO” is determined in the stepS213 or S215, the process directly proceeds to a step S219 while if“YES” is determined in both of the steps S213 and S215, the count valueGcnt is incremented in a step S217 and then, the process proceeds to astep S219. Accordingly, the count value Gcnt is incremented when thecolor evaluation value at the noticed divided area belongs to the areaG, and the image in the divided area has a high luminance.

[0094] It is determined whether or not a determination process withrespect to all the divided areas shown in FIG. 18 is performed in thestep S219, and the processes in the steps S203 to S217 are repeateduntil “YES” is determined. Consequently, the count value Rcnt is renewedaccording to a determination result of the steps S207 and S211, and thecount value Gcnt is renewed according to a determination result of thesteps S213 and S215.

[0095] The area R shown in FIG. 19 is the area including a large amountof R component, and the area G is the area including a large amount of Gcomponent. Accordingly, the more the section having the high luminanceand including the R component is, the larger the count value Rcnt is,and the more the section having the high luminance and including the Gcomponent is, the larger the count value Gcnt is. Furthermore, asunderstood from FIG. 19, the area R also includes the color of the skinof the person. Therefore, when photographing the face of the person at abright place, the count value Rcnt is incremented at a time ofdetermining a divided area in which an image of the face exists.

[0096] Each of the Rsat, the Gsat and the Bsat is detected withreference to the table 44 a in a step S221. As described above, the Rsatis the color saturation degree relating to the R, and the Gsat is thecolor saturation degree relating to the G. Furthermore, the Bsat is thetotal number of pixels in which the B level is saturated and can bedefined as the color saturation degree relating to the B.

[0097] The count value Gcnt is determined in a step S223, and the Gsatis compared with the Bsat in a step S225. If a condition of Gcnt=0 issatisfied, or if a condition of Gsat≦Bsat is satisfied even if Gcnt≧1,the process proceeds to a step S227. On the other hand, if Gcnt≧=1 andGsat>Bsat, the process proceeds to a step S229. The Rsat_N is calculatedaccording to an equation 5 in the step S227, and the Rsat_N iscalculated according to an equation 6 in the step S229.

Rsat_(—) N=Rsat−Gsat  [equation 5]

Rsat_(—) N=Rsat−Bsat  [equation 6]

[0098] As described above, generally, when the G signal is saturated,the luminance level is also saturated, and therefore, the Gsat isdefined as the luminance saturation degree in FIG. 1 embodiment. Then,by subtracting the Gsat from the Rsat, the total number of pixels inwhich the luminance level is not saturated but the R level is saturatedis obtained. However, with respect to some objects, the luminance levelis not saturated while the G level is saturated, and in such the object,the Gsat cannot be defined as the luminance saturation degree. On theother hand, when the luminance level is saturated, not only the G levelbut also the B level is saturated, and therefore, a smaller numericalvalue out of the Gsat and the Bsat approximates to the number of pixelsin which the luminance level is saturated. Therefore, by comparing theGsat and the Bsat with each other in the step S225 and subtracting thesmaller numerical value from the Rsat, the Rsat_N being the deviationdegree of the R signal is acquired.

[0099] Furthermore, when Gcnt=0, the G component is not included in theobject, or luminance of the object is entirely low. Therefore, inaccordance with a principal, the Gsat is defined as the luminancesaturation degree. At this time, the process proceeds to the step S227without performing a comparison process in the step S225, and the Rsat_Nis obtained according to the equation 5.

[0100] A value of the calculated Rsat_N is determined in a step S231.Then, if a condition of Rsat_N>0 is satisfied, the process directlyproceeds to a step S235, and if a condition of Rsat_N≦0 is satisfied,“0” is set to the Rsat_N in a step S233, and then, the process proceedsto the step S235. The processes are the same as that in the steps S87and S89 shown in FIG. 7.

[0101] The count value Rcnt is compared with “1” in the step S235. Then,if a condition of Rcnt≧1 is satisfied, the Rsat_N is renewed accordingto an equation 7 in a step S237, and if a condition of Rcnt=0 issatisfied, “0” is set to the Rsat_N in a step S239.

Rsat_(—) N=Rsat_(—) N*Rcnt  [equation 7]

[0102] When the Rsat_N is determined, processes the same as theprocesses in the steps S91 to S95 in FIG. 7 are performed in steps S241to S245. Thus, the reference value Ys is determined.

[0103] Since a general user is apt to select a person as the object, asituation in which a distortion of a hue occurs in an image of the skinof the person has to be avoided. Therefore, the R component mostincluded in the image of the skin of the person is noticed, and when thecondition of Rcnt≧1 is satisfied, the Rsat_N is multiplied by the Rcnt.That is, the deviation degree of the R is weighed depending upon thenumber of the divided areas which has a high luminance and belongs tothe R area. Thus, the larger the count value Rcnt is, the lower thereference value Ys is, and the short-time exposure image signal becomeseasy to be selected by the switch SW1 shown in FIG. 9. Consequently, thedistortion of the hue with respect to a color including the R componentis controlled.

[0104] It is noted that when Rcnt=0, “0” is set to the Rsat_N, andwhereby, the reference value Ys is set to the maximum value Ysmax. Thecount value Rcnt becomes “0” when the images of all the divided areashave a low luminance or never include the R component. In such the case,the hue is probably not distorted even if the reference value Ys is setto the maximum value Ysmax, and therefore, the reference value Ys is setto the maximum value Ysmax.

[0105] Although the present invention has been described and illustratedin detail, it is clearly understood that the same is by way ofillustration and example only and is not to be taken by way oflimitation, the spirit and scope of the present invention being limitedonly by the terms of the appended claims.

1. An image combining apparatus which generates, on the basis of a firstimage signal of an object obtained by a first exposure according to afirst exposure amount and a second image signal of said object obtainedby a second exposure according to a second exposure amount which is lessthan said first exposure amount, a combined image signal of said object,comprising: a comparing means for comparing a brightness relating levelof any one of said first image signal and said second image signal witha reference value; a first selecting means for selecting said firstimage signal when said brightness relating level is equal to or lessthan said reference value; a second selecting means for selecting saidsecond image signal when said brightness relating level is larger thansaid reference value; a deviation degree detecting means for detecting adeviation degree of color of said object; and a reducing means forreducing said reference value on the basis of said deviation degree. 2.An image combining apparatus according to claim 1, further comprising afetching means for fetching a third image signal of said object obtainedby an exposure according to a predetermined exposure amount, whereinsaid deviation degree detecting means includes a color saturation degreedetecting means for detecting a color saturation degree of a specificcolor based on said third image signal, a luminance saturation degreedetecting means for detecting a luminance saturation degree of saidthird image signal and a subtracting means for subtracting saidluminance saturation degree from said color saturation degree.
 3. Animage combining apparatus according to claim 2, wherein said colorsaturation degree detecting means detects a first number of pixels inwhich a color level of said specific color is saturated, and saidluminance saturation degree detecting means detects a second number ofpixels in which a luminance is saturated.
 4. An image combiningapparatus according to claim 2, wherein said predetermined exposureamount is less than said first exposure amount and more than said secondexposure amount.
 5. An image combining apparatus according to claim 1,wherein said reducing means greatly reduces said reference value as saiddeviation degree is large.
 6. An image combining apparatus according toclaim 1, further comprising a determining means for determining whetheror not each of a plurality of sections firming an object image satisfiesa predetermined condition, and wherein said reducing means includes aweighting means for weighting the deviation degree depending upon thenumber of sections satisfying the predetermined condition and areference value reducing means for reducing said reference value on thebasis of a result of the weighting by said weighting means.
 7. An imagecombining apparatus according to claim 6, wherein the predeterminedcondition includes a first condition indicating that a noticed sectionis a specific color and a second condition indicating that said noticedsection has a high luminance.
 8. A digital camera provided with an imagecombining apparatus according to claim
 1. 9. An image combining methodwhich generates, on the basis of a first image signal of an objectobtained by a first exposure according to a first exposure amount and asecond image signal of said object obtained by a second exposureaccording to a second exposure amount which is less than said firstexposure amount, a combined image signal of said object, comprisingfollowing steps of: (a) comparing a brightness relating level of any oneof said first image signal and said second image signal with a referencevalue; (b) selecting said first image signal when said brightnessrelating level is equal to or less than said reference value; (c)selecting said second image signal when said brightness relating levelis larger than said reference value; (d) detecting a deviation degree ofcolor of said object; and (e) reducing said reference value on the basisof said deviation degree.
 10. An image combining method according toclaim 9, further comprising a step (f) of fetching a third image signalof said object obtained by an exposure according to a predeterminedexposure amount, wherein said step (d) includes a step (d-1) ofdetecting a color saturation degree of a specific color on the basis ofsaid third image signal, a step (d-2) of detecting a luminancesaturation degree of said third image signal and a step (d-3) ofsubtracting said luminance saturation degree from said color saturationdegree.
 11. An image combining apparatus according to claim 3, whereinsaid predetermined exposure amount is less than said first exposureamount and more than said second exposure amount.
 12. An image combiningapparatus according to claim 2, wherein said reducing means greatlyreduces said reference value as said deviation degree is large.
 13. Animage combining apparatus according to claim 3, wherein said reducingmeans greatly reduces said reference value as said deviation degree islarge.
 14. An image combining apparatus according to claim 4, whereinsaid reducing means greatly reduces said reference value as saiddeviation degree is large.
 15. An image combining apparatus according toclaim 2, further comprising a determining means for determining whetheror not each of a plurality of sections forming an object image satisfiesa predetermined condition, and wherein said reducing means includes aweighting means for weighting the deviation degree depending upon thenumber of sections satisfying the predetermined condition and areference value reducing means for reducing said reference value on thebasis of a result of the weighting by said weighting means.
 16. An imagecombining apparatus according to claim 3, further comprising adetermining means for determining whether or not each of a plurality ofsections forming an object image satisfies a predetermined condition,and wherein said reducing means includes a weighting means for weightingthe deviation degree depending upon the number of sections satisfyingthe predetermined condition and a reference value reducing means forreducing said reference value on the basis of a result of the weightingby said weighting means.
 17. An image combining apparatus according toclaim 4, further comprising a determining means for determining whetheror not each of a plurality of sections forming an object image satisfiesa predetermined condition, and wherein said reducing means includes aweighting means for weighting the deviation degree depending upon thenumber of sections satisfying the predetermined condition and areference value reducing means for reducing said reference value on thebasis of a result of the weighting by said weighting means.
 18. Adigital camera provided with an image combining apparatus according toclaim
 2. 19. A digital camera provided with an image combining apparatusaccording to claim
 3. 20. A digital camera provided with an imagecombining apparatus according to claim 4.